Expert Elicitation of Adversary Preferences Using Ordinal Judgments

Obtaining probabilistic judgments from subject-matter experts is a difficult task, especially when those experts are not quantitatively trained. In this paper, we introduce a simple elicitation process where intelligence experts express their knowledge about adversary preferences by rank ordering the attractiveness of a collection of possible terrorist targets (or attack strategies), where the attractiveness or utility of each target to the terrorist(s) is assumed to involve multiple attributes. The probability distributions over the various attribute weights are then mathematically derived using either probabilistic inversion (PI) or Bayesian density estimation (BDE). This elicitation process reduces the burden involved in traditional methods of attribute-weight elicitation, and explicitly captures the existing uncertainty and disagreement among experts, rather than attempting to achieve a potentially misleading consensus. This work also makes broader methodological contributions to the fields of utility assessment and expert elicitation, by allowing the use of "unobserved attributes" to ensure the feasibility of PI, by showing how to apply BDE to ordinal data in a rigorous manner, and by elucidating the relationship between PI and BDE.